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This report documents (1) the problems inherent in 
multiple choice testing, (2) a solution to the problems and (3) 
computer programs required by the solution. Problems of multiple 
choice testing include scheduling inflexibility, methodological 
inflexibility, cheating, inefficiencies of space and student 
interaction time, inefficiencies of instructor preparation time, 
uneven quality of examinations, and expense of preparation. The 
solution has four aajor features: (1) a special item file design, (2) 
a method of generating a large number of unique, equivalent measure 
tests, (3) published item files, and (4) a testing station 
independent of the classroom. A description and analysis of the 
practical implementation are also given. The implementation was 
highly regarded by the students (92 percent preference) as they 
believed that they learned more and performed better under this than 
under standard testing procedures. (HV) 
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Abstract 



&BSTHACT 



This report documents (1) the problems inherent in 
multiple choice testing, (2) a solution to the problems, and 
(3) computer programs required by the solution. It also 
describes and analyses a partial implementation of the 
solution. The implementation was highly regarded by the 
stud9nts (93% preference) as they believed that they learned 
more and performed better under it than under standard 
testing procedures. 
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PROBLEM 



Each quarter at Virginia Tech over 40,000 multiple- 
choice test sheets are processed by the Learning Resources 
Center. This number has been increasing at the rate of 10 
to 20 percent per year since the service was established in 
1972 and will probably exceed 50,000 sheets this quarter. 
For the most part, this testing . is concentrated in a 
relatively small number of undergraduate courses with class 
sizes in excess of 100, 

Under these circumstances use of machine processing 
followed by computer analysis and score reporting is 
virtually unavoidable. Given the resources available to 
faculty in charge of larga sections, some with o^.rer 500 
students, other methods of resting are simply too time 
consuming. Also^ there is a further problem of maintaining 
uniformity of scoring when this work is distributed over a 
number of graduate assistants. 

Quality and efficiency of multiple-choice testing is 
dependent upon a number of factors. However, the standard 
method of creating and administering multiple choice 
tests (A) has inherent deficiencies. A nonexhaustive list of 
deficiencies follows: 



SCHEDULING INFLEXIBILITY 

Students often miss tests, seek rescheduled tests, or 
perform poorly on tests for legitimate reasons. Unavoidable 
yet transitory personal problems and multiple tests 
scheduled on the same day in several courses are two of the 
more obvious reasons. in any case, either the conscientious 
instructor must assume extra administrative burden (validity 
of reasons often must be verified and alternative testing 
procedures must be maintained) or the hapless student must 



(A) The standard method of multiple choice testing at VPI 
is: (la) an in struct o r-- typist interaction generates copies 
of a test or copies of scrambled versions of a test, or (lb) 
a computer program generat-^o tests by selecting fixed items 
(each selection of an item produces the same information) 
from a file of fixed items, and (2) the tests are given to 
students at an assigned time and place (s). 



Individualized Testing System Calhoup 5 

Problem cent 



suffer. 



METHODOLOGICAL INFLEXI^LlTY 

Retesting require^ by mastery learning techniques and 
various other administ tative and educational techniques can 
not be readily implemented ^ith present testing systems. 



CHEATING 

While cheating by Copying answers of an adjacent 
examinee has uct rec^iv^d the press coverage of some other 
forms of cheating, th^ PJrevalence of this phenomenon has 
been documented in the University environment (B) , Cheating 
on-.tests can be controilea by use of multiple forms of a 
testy careful control of seating arrangements, and rigid 
proctoring; but many itiii^tructors are unwilling to take such 
strenuous measures beea^s^ they believe that student- 
instructor rapport will be adversely affected. 



li|IIieiSNCi;ES OF SPA^ AND STUPEKT'INSTRQCTag INTERACTION 

Each class is allocated resources, principally in the 
form of X seat hoars and Y instructor hours. The limits of 
student-instructor interaction are fixed by this allocationT 
Inclass^ testing consumes a significant portion of this 
allocation. For examjiie, the typical lecture based course 
at VPT (3 contact houir^ P^r week for 9-10 weeks) requires 
2-3 hours for inclass t^sts (excluding finals). That is^ 
7-11% of the allocated i^^sources are consumed by tests. 
This use of time is not inconsequential as studen t-intructor 
interaction could be inc^i^^ased by that amount (presumably 
enhancing the student's education) or the student's stay at 
VPI could be reduced by that amount (saving the tax payers 
of Virginia considerable sums of money) if tests were 
removed from the classroom. For example^ the equivalent of 
the YPI Introductory Biology series (3 contact hours for 3 
quarters) could be taught as two^ U hour courses. 



(E) Frary^ R. 3 and T. N. Tideman^ Detection of Remarkable 
Similarity Between MultiPle-choice Test Responses. Paper 
presented at annual -meeting of American Psychological 
Association^ Chicago, September, 1975. 
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IiiiEFFI£IENClKS OF INSTRUCTOR £ilEPARATION TIME AND HJ^EVEN 

QUhkllX 21 Mami NATIONS " " 

While tests are among th^ most powerful ediii;atioaal 
tools, preparation of each test is a creative effoirt vhich 
competes ^ith research, adaiinistrati ve duties, lecture 
preparatioHr ate, for the instructor's time and en^^gy^ 
Since this competition occurs several times duti^^^ the 
academic year, test preparation often comes out on ti^e ^hort 
end. Pushed for rime, the instructor hurriedly c^^^tes a 
low quality test from scratch oi: uses essentially a previous 
test {which may be last quarter's last minute" t^st) . 
Education is poorly served in either case. Low quality 
tests frustrate the more conscientious students 5c?r these 
students are penalized relative to weaker students ^ince 
chance pl^ys a larger role in poor tests. Previous tests 
are a better measure of the students "connection^* than 
knowledge. 

While some instructors maKS more of their efforts in 
this area than others, no instructor at VPI can readily 
analyse the quality and results of his tests i^ d^^tai?. 
simply because the necess^ary use-statistics on t^^t Items 
are not r-^adily available. 'Careful analysis of th^ results 
of tests is absolutely necessary iri directing the instructor 
toward improvements in both his Presentation and hi^ testing 
procedure* 

Uneven quality is also a Problem with most c^omputer 
based syste^is due to poor organization, lack of compactness 
of information, or lack of analytical features. 



The process of having a secretary type a te^t, the 
instructor proofread the test, and the secretary retYP^ the 
test is tiin^ consuining and er-or prone. The investment 
increases tremendously if more than one form (sc^ranjbled 
version) of a test is reguire^. While this expanse is 
probably reduced by all computer based systems, the savings 
vary from system to system depending on design. 
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SOLUTION 



A solution has b^en found (and in pajrt demonstrated) 
that has U major features. The features ^i^e (1) a special 
item file design^ (2) ^ method of genera'ting a large number 
Of unique^ equivalent measure tesrs^ (3) published item 
files, and (^i) a testing station i^iciependent of the 
classroom. Each feature of the solution is discussed in 
turn. 



S2SIGN 

The itea file was designed (1) to aJ^lov a large number 
Of true and false responses for each itein '^tem' (i.e., the 
question portion of the itec) ^ (2) to all<pw the inclusion of 
outlxne material, and (3) to allow the in^l^^ion of response 
Use-statistics. u^e of these design features aid the 
instructor in creating, organizing^ and evaluating his test 
Items. They also aid the instructor in evaluating his 
presenrations and the student's grasp the study areas* 

Each design feature is discussed in turn, 

^S£,a,§, ^iifiber of .Trug, aiid False Hes£onses ^r item Stem 

Through the association of a large r^^niber of respons.es 
with each stem, the instructor reduc;es the need for 
redundant information, and the resultant c;oncise information 
is much easier to access and evaluate- This multiple 
association also len^is itself to efficient item creation as 
the major concept is usually embodied in the stem. 

Outlijoe Ma terial 

Outline material and comments are easily placed in the 
item file and are necessary for proper o^^g^nization of an 
item file* A highly organized item file is absolutely 
necessary for even evaluation of both th^ student's grasp 
and the instructor's effectiveness. In fact^ the item file 
should usually be more highly structured than other 
materials (e.g.r lectures and text books) because it usually 
carries the same range of ideas as the presented material 
but at various levels of difficulty. 
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Use-S tatisti cs 

Although this fpdture i^ not available at present, it 
will soon be available ^nd thus is discussed for 
coapleteuess. Analysis viH be accomplished by 

incorporating use-statistics each response into the item 
file. These statistics wiil allow the instructor to 
deteriaine vhich type of student missed the response and 
which type of student di^ not* These statistics are 
absolutely necessary if the instructor desires to measure 
his effectiveness, to measure the students grasp, and to 
measure the response's validity. of course the 3 measures 
are confounded and no -statistic will substitute for the 
instructor's experience "ana g^od judgement in deciding which 
measure is having the greater influence. But the fact 
remains that at present no in^^truc; tor at VPI has access to 
the information needed to maJ^e that judgement. 

In closing this section qU the design of the item file, 
I hasten to restate rhat the item file is designed to allow 
organization and analysis. It encourages these 

characteristics by making th^m easily available, not by 
making them mandatory. In fact, even standard items (i.e., 
fixed items) can be entered into the item file with less 
effort than is customarily used in typing a test. This 
apparent laxity is purposeful, serving the larger role of 
allowing instructors to use th^ item file as if it were a 
standard item file. Thus, the instructor can grow into full 
use of the item file at his peculiar pace. 



MSTHOD OF GSNERATI NG 0NI2IJH^ E^iiiiALSNT K2A5URE BXAKS 

AS lony as copies of only one test are available for 
use by a class during a testing period (e.g., a midterm) 
inflexibility must be the r^l^ and cheating will be a 
correlary. That is, everyone involved must be at designated 
locations at a specific time, and in all probability someone 
will look at a neighbor's test* But the customary use of 
only one test results from administrative problems rather 
lhan conceptional ones, Aftei: all, with tests we are 
sampling the student's grasp of a study area through the use 
of a small sample of items. This small sample of items 
comes from the potentially infinite set of items stored in 
the instructor's brain. If an instructor could easily and 
randomly sample that set of items, he could generate a set 
of unique^ yet equivalent measure tests. By unique, I mean 
that the tests would be Sufficiently different that a 
student could not gain advan^'^ag^ by studying one of the 
test.s, e.g.^ one taken by a fjriend, prior to taking another 
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of rhe tests for credit. By equivalent measura^ I mean that 
any two tests from the set would provide equal, ataasures of 
the student's grasp of the study area. With a set of 
unique^ equivalent measure tests on a study area^ the 
instructor could be quite flexible in administering tests 
and could be properly unconcerned with the common forms of 
cheating. 

Thusr ^ part of the solution is a method of generating 
unique, equivalent measure tests. This is accomolished by 
randomly sslacting a specified number of items from 
sp^cifiAi ^ryas of an item file (areas corresponding to 
study areas) and randomly selecting true and false responses 
for each selected item. Equivalent measure of each test 
results from homogeneous areas of items and from the 
smoothing effect of random selection. Uniqueness results 
froa the size ot the itea file* Given the compact design of 
the item ftle^ the number of items needed for uniqueness of 
generated tests is well within the reach of an instructor* 



A -efficiently large (too large for a student* to 
memorize), published item file would serve the immediate 
educational purpose of directing the student's study efforts 
and would reduce the administrative efforts required to 
maintain test security and to ensure equitable distribution 
of old tests (koofers at VPI) to all students. 



IS^TING STATION INDEPENDENT OF THE CLASS KOOM 

A tasting station independent of the classroom is 
necessary for efficiency of space and for flexibility. The 
testing station should consist of a comfortable room and a 
friendly human monitor. The monitor would check IDs^ 
dispense appropriate tests to students as they enter the 
room, and dispense test keys to students who have completed 
their tests, providing the tested students with instant 
feedback* The station should be open at convenient times* 
Thus, a student would be free of time and other petty 
pressures that often adversely affect his performance. 
Operation of the station would provide economies of scale 
and skill. That is, many classes could use the same 
station, and the monitor's required skills would be minimal. 
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Certain computer programs are ' necessairy for 
iinpieaeataticn of the solution* I have written three 
prograas and a fourth remains to do written. The 3 existing 
programs are called ITSPJlT, ITSANL, and IT3GSN, ITSOPD 
remains to be written. Each progiam is described in turn. 



ITSFMT 

ITSPMT reads through a raw (changed or recently 
created) fixe producing (1) an it-em file suitable for use by 
the other programs and (2) a statistics listing. This 
program allows the secretary to put less effort into 
creatiing or changing an item file than he would put into 
typing a test because ITSPMT does most of the work in 
for-riatting and because th^ statistics listing, with its 
descriptive statistics and error messages, makes detection 
and correction of logical errors easy. Figure 1 illustrates 
a small, raw file called FIGU ITSFMT operated on FIG1 to 
produce item file ?IG2 of figure 2 and simultaneously the 
statistics listing of figure 3. The remainder of this 
section .is a detailed description of the raw file, 
statistics listing, and item file. 

Raw Item File 

The items erterevi in the raw file by the secretary have 
minimal format. The item ID is entered as one record, 
followed on subsequent records by the stem. Next the 
response ID is entered as one record, followed by the 
response on subsequent records. Other responses are entered 
in a like manner. The only other restrictions on entry of 
an item are that (1) records entered into the raw file must 
be less than 256 characters in length, (2) an item (stem + 
responses) and its comments must occupy less than 32,768 
records in the itT a file where the maximum record length is 
80 characters (longer records in the raw file are broken at 
word boundaries, the extra length being inserted as a new 
record) , (3) an item must have a stem and at least 1 true 
and 1 false response, and (U) an item must be left justified 
at the 2nd column. 

Column 1 is reserved for special characters. An item 
ID record must iiave a in column 1 while a response ID 

record must have a or »-< depending on the resDonse's 
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truth value. An item ID may be any number of the fora 
• ######?^, #^###### » where •#« is a decimal digit (leading and 
Trailing zeroes are not necessary) . This range is provided 
ro allow stability of old item IDs while allowing for 
insertion of new irems and to allow latitude in choice of a 
systematic iiaabering sysrem if one is desired. a response 
ID aust he integers from 1 through 32,767. Both the item 
IDs and the response IDs (within an item) must be strictly 
raonotonically increasing. Records with a '.'in column 1 
and preceding a response ID record are considered comments. 
5i.-ailar records preceding an item ID record are considered 
ourline material. The records never appear on a test; 

they are strictly notes for the instructor. 

^S^iistics Listing 

If any logical errors exist in the raw file, ITSFMT 
will correct them or ignore the offending items. in either 
case, there will be no logical errors in the item file even 
though it may have fewer items than intended by the 
instructor. Thus, to aid the secretary in catching and 
correcting logical errors, both the type and location of 
logical ^ errors in the raw file are documented in the 
statistics listing along with useful descriptive statistics. 
Exaniples of logical errors are improper numbering of items 
or responses, nonspecip-1 characters in column 1, no 
responses for an item, etc. The statistics listing of 
figure 3 shows that raw file FIG1 has no logical errors. It 
also shows that the item beginning on record 3 of FIGT has 3 
more true responses and 5 more false responses indicated on 
the item ID record of FIG2 than indicated on the item ID 
record of FIG1. For the item beginning on the 24th record 
the numbers are 5 and 7. Both files have 53 records of 2 
items. 

Item File 

An item file is an exact copy of a raw file (assuming 
no logical errors in the raw file) except (1) all records 
have been reduced to BO characters or less and (2) 
additional fields may have bean added and initilized. An 
item file has 3 fields on each item ID record that carry 
i-em descriptive numbers that are primarily for use by the 
other programs. These are the number of true and false 
responses^ and the number of records occupied by the item 
and its comments. For example, in item file FIG2, item 
139.557, the numbers are 3, 5, and 22, respectively. In 
addition to the item descriptive fields, several fields are 
set aside by ITSFMT to carry use-statistics once they become 
availabl'.^. These fields are initilized to •0« if they are 
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undefined in the raw file. Each item ID record has a field 
reserved for the number of times the item appeared on a 
testr and each response ID record has 4 fields reserved for 
(1),the number of times a student reacted properly to the 
response (i.e.^ marked true responses and left false 
responses unmarked)^ (2) the number of times students 
reacted improperly to the response^ (3) the sura of the 
standardized test scores of the properly reacting students^ 
and (4) the sum of the standardized test scores of the 
improperly reacting students. 

After items have been used on tests^ the said use- 
statistics would be accumulated from the students answer 
sheets and added to the item file. ITSUPD would handle this 
chore and will be discussed later. However^ it serves my 
presentation to show how item file FIG2 might appear with 
use-statistics. Figure 4 is item file FIG2 with simulated 
use-statistics. This file is now called FIG4. 

In closing this section on ITSFMT, I restate that 
ITSFMT allows a secretary to create an item file with less 
effort and in less time than is required to type a test. 
This detailed discussion is presented for documentation of 
the program and is not necessary knowledge for its use. In 
fact, the structure of the entered item (the only one the 
secretary has to understand) is sufficiently intuitive that 
few errors will be made and the most likely ones will be 
corrected by the program. ITSFMT is easily operated. It 
needs to know where the raw file is located and where rhe 
item file is to be placed. The statistics listing is 
written on SYSPRINT. 



ITSANL 

Even though "'the item file is quite attractive to the 
computer programs and ensures execution efficiencies^ an 
exact listing of the item file is definately not for human 
use. The tremendous detail of an exact listing prevents an 
overview of the item file^ and the cumulative statistics 
prove awkward in analysis. Thus^ ITSANL was written to 
provide two listings for the instructor: (1) a contents 
listing that contains outline material and item IDs and (2) 
an analytical listing that ■ contains easily understood 
statistics instead of the cumulative statistics. Figures 5 
and 7 were simultaneously produced by ITSANL from item file 
FIGa. Figure '5 is the contents listing and, figure 7 is the 
analytical listing. Howe ver , since item file FIG a is so 
small that an overview is obvious, to carry the point I 
provide a contents listing from a larger item file (figure 
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6). 

From tlie analytical listing the instructor can learn 
for each response (1) the number of times it appeared on a 
test^ (2) the percentage of students who reacted properly to 
it (i.e., mark-T tr response or left a false response 
unmarked), ige standardizad score of the proper 

reactors, average standardized re of the 

improper r , example, response 1 o em 139.557 

(figure 7) > 424 tests; 20% of the st ucs reacted 

properly to xt; the average standardized score of the 20% 
was 79; and that of the other 80% was 45. Through careful 
reflection on this analytical information the instructor can 
measure his performance in each area of presentation, the 
students' grasp of each area, and each response's validity. 

One should contrast the ease of understanding the 
information carried' in figure 7 with that of figure a. In 
most cases, the instructor would have no reason to see exact 
listings of his item file, and he would work wirh contents 
and analytical listings only. 

ITSANL is easily operated. It needs to know where the 
item file is located and where the contents and analytical 
listing are to be placed. 



ITSGEN 

ITSGEN generates (1) a specifications listing and (2) a 
set of tests. The specifications listing lists the 
specifications used in generating the tests and any errors 
encountered. 



The instructor must provide ITSGEN with one group of 
specifications for each set of tests to be generated. Three 
types of specifications are required for generation of a set 
of tests; other specifications are available to allow 
greater control over the generation of a set of tests and to 
allow for the production of auxiliary information. 
Specifications are of the form 'keyword = value;*, and the 
last specification of a group must be followed by «END;». 
Each specification must be less than 32,763 characters in 
length. Table 1 is a list of all keywords, their use, and 
default values. The tests of figure 10 and the 
specifications listing of figure 9 resulted from the 
operation of ITSGEN on the group of specifications in figure 
8. A discussion of this example will serve to illustrate- 
the use of specifications. 
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Two selections ('SELECTIONS = 2;») were specified. A 
selection is a test generated by randomly picking items from 
the pool of specified items and randomly picking responses 
from the chosen items. In this example, two items were 
ran^domly picked from the items with IDs between 100 and 200^ 
inclusively ('REQUEST ^ 2 100 - 200;'), and, by default, up 
to 1 true and 4 false responses were randomly picked from 
each chosen item. Thus, each selection was a unique test. 
Two versions ('^'^RSin-- =2;M of each selection were 
specified. A ve^:ii(j. . test generated by randomizing the 
order of each item's chosen responses. Thus, the two 
versions of^ each sel .l. ion had the same items and responses 
but the responses were scrambled. Most instructors would 
not ask for both versions and selections. Many selections 
and 1 version (multiple tests) would be the rule for 
individualized testing, and many v3rsions and 1 selection 
(multiple forms) would be the rule for inclass testing. I 
specified both to show the generality of ITSGEN. The tests 
wer^ 80 characters wide ( • LINESIZE = 80;») and had a title 
on the first page of each test ('TITLE = SPECIFICATION 
EXAMPLE;'). Item file PIGU contained the. requested items 
(•ITEM^FILE = FIGU;'), and the generated tests were written 
on file FIG10 '('TEST^FILF = FIGIO;*), 

Figure 11 is another example of specifications used to 
generate a test. Figure 12 is the specifications listing, 
figure 13 the test, and figure lU the test key. In this 
example, a item files were searched for 18 peculiar items. 
This is a realistic example as 2U0 copies of the generated 
test and - key were printed and handed" out to students as a 
study guide. This effort cost about 10 computer dollars and 
10-20 human minutes even though no special effort was made 
toward efficiency, 

A specification listing has up to U sections. one is 
essentially a relisting of the specifications. Any logical 
error in the specifications would be caught here, an 
appropriate error message printed, and execution on that 
group of specifications would cease. The second section 
lists all specifications obtained. The third section lists 
the items requested ("POSSIBLE ITEMS') in a sorted, parsed 
form. Any logical errors in an item request would be caught 
at this point, an appropriate error, message printed, and 
execution on that group of specifications would cease. The 
fourth section lists the number of requested and available 
items. Should insufficient items be available, an error 
message would be printed and execution on that group of 
specifications would cease. 

ITSGEN is easily operated. It needs to know where the 
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item files and specifications are located and where the 
tests and requested auxiliary information are to be placed.-^'" 



ITSUPD 

This program is not completed. It will combine the 
information in the analysis file (an auxiliary information 
file that carries each item and response ID of each 
generated test) with the students' test results and compute 
use-st- ' ' ';tics. The results will be used to update the use- 
stati of the item files. 



16 



Individualized Testing System Calhoun 16 

Implementation 



IMPLEMENTATION OF SOLUTION 



While the solution seemed logical, I realized that 
■logical' solutions do not necessarily function as expected. 
Thus, to demonstrate its feasibility and to gain operational 
information,, the solution was implemented on a small scale. 
This implementation is labeled ITS (individualized testing 
system) . 



Four instructors in 4 different classes were involved 
in this implementation of the solution (Fall 1975 and Winter 
1976 Introductory Physiology and Winter 1976 and Spring . 1976 
Introductory Genetics with 173, 95, 163, and 97 students, 
respectively) . Two major compromises were necessary. 

(1) The solution requires a testing station available 
at convenient hours, e.g., those of a library. This would 
allow the students wide latitude in scheduling their tests 
and would allow the instructor to test and retest in any 
pattern or frequency desired. However, the scale an^ 
resources were too small tc justify operating the testinc 
station as envisioned, Thu -he iastrructors agreed to use: 
only one pattern of testin- This was essentially s test- 
retest pattern at standard t ing periods. a studen-: coiuld 
take 1 test during the fir week and 1 test dar±ng -he 
second week of a resting pet.od. If the student trxok both 
tests, only the second test ^^test) counted (fleets v^xe 
exceptions in 2 of the class as the maximum of te^rt-- 
retest counted) . Grades were osted after the first w^ek so 
that the, students could make informed decisions on i n- a 
second test (2 classes posted letter grades, 1 clas^: -^ostred 
scores, and 1 class posted adjusted scores). The testing 
station was generally open 12 hours a week (3 hour^ in the 
evening on each of 4 consecutive days) . 

(2) Th^ isolution require^: item files sufficiently large 
for publics-ion (too large for memorization). ' These 
publications would aid a s::i^xdent in directing his study 
efforts and would antiquate jrost efforts undertaken in the 
name of tes-^ security and Equitable distribution of old 
tests. liowever, sufficient icem files were not available. 
Thus, as a study guide, copies of. a test and its key were 
distributed a week before the testing period. The security 
of the item files was maintained by retaining all tests at 



17 



Individualized Testing System Calhoun 17 

Implementation cont 



the testing station. 

A prototype of ITSGEN was us^d to generate sets of 
unique tests for each testing perioti. Tvro instructors 
created iteia files from scratch while the other 2 modified 
existing item files. Three of the daises used the system 
for all testing periods: midterms^ endterms, and finals. 
One of the classes used it only for its endterm^ using essay 
tests for the midterm and final. aH students were given 
instructions on the use of the testing J^tation (figure 15) a 
week before the first testing period- The protocol at the 
testing station was: the incoming student's ID was checked 
against a class roll and he was giv^n the appropriate test, 
the tested student exchanged his answer sheet for the key to 
his te' and studied his test and key^ and the departing 
student returned his test' and key to the monitor. 



ANALYSIS 

While the solution is very broad^ its was necessarily 
tentative and narrow. Thus^ an analysis of all aspects of 
t i ? 3;:lation Ls not possible from IJS (mosl: benefits of the 
ol-z—±cn are salf^evident anyway). However, from student's 
ir^ersy grades^ and conversations* an analysis of the 
- iuca-nional ef±ects of some aspects of th^ solution can be 
--dd-^; nnd from experience with this implementation^ I can 
c:c:tin:.:^.rt: on cost effectiveness. 



Szudents in 3 of the classes we^e surveyed afeer their 

zrixr r ITS te:StirLg period. Figure 16 is the suxvsf and 
-:rif:3xil t3 . The numbers preceding the responses ax^e the 
i:i-ercent:age (rounded to integers) of the students who 

Tair-3idiexed the xnem and marked that response. The three 
represent the 3 classes sur^^yedr and the classes 

ir^ rererenced 'left', 'middle', and 'right' depending on 

:he p:^sition of their column. A »_« means that the response 
nor on t±i9 survey given to tha^ class. The surveys 

"cri>>ably had nhe following validity relationship: 'r±ghrt' > 

*i2i151e' > 'left'. The : rimary reasons for this ran-king 
v-7rtr 1) the number and per::antage of students responding to 

-...a surveys {62% of 91, 'Is:-'; 65% of 'middle'; and 76% 

c:f 1o3, 'right') and (2) t: 3 propriety of -:he test items. 

^ .-: instructor of the 'right' class u^ed bis own item files^ 
^:3y closely paralled his lectures; the instructors of 
-iL-Tf 'aiiddle' class and 'laft' clas^ primarily used , item 
file- -reated. by others^ and they encountered and coped with 
diT^. - ice of lectures and :item fileS to different degrees 
— tme instrucT:c;r of the 'middle' class was more active and 
3 -iccessf ul. 
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Impropriety of the test items given to the 'left* class 
is further supported by the fact that the 'left* class 
performed much poorer at a comparable testing period than 
the 'right' even though essentially the same item files were 
used to generate the tests for both classes (mean = 60.6, 
s.e. = 1.62 for 'right* and mean = 50.8, s.e. = 1.84 for 
■left'). In addition, from conversations with students, I 
believe that students of the 'left' class were led to blame 
the 'computer' for the impropriety of their test items. 
Thus, I restrict my discussion of the survey to the 'right' 
class, though there was general agreement of all three 
surveys 

Survey comments (item 23) were divided into 3 classes: 
positive (figure 17), negative (figure 18), and irrelevant. 
Comments considered irrelevant were "I like the course but 
the book is terrible.", "Large computer sheet was very nice 
to have.", and statements on imaginary, corrected, or 
uncorrectable technical problems. 



Educational Aspects^ Positive 

The reaction to ITS v/as so overwhelmingly positive that 
a sophisticated analysis was not required. students were 
pleased witn ITS because they believed that they learned 
iaore and performed better under it than under standard 
testing procedures. This opinion was independent of the 
pattern of testing (test- re test) since 80% of the students 
(item 19) would have perferred ITS to inclass testing even 
if retesting were not available. The benefits offered by 
the testing station were probably responsible for this 
acceptance. That is, 67% (item 13) thought choosing their 
test day and 81% (item 17) thought the atmosphere of the 
testing station contributed positively toward their 
performance. Even though 80% is a high acceptance rate^ it 
is probably a low estimate of a true acceptiance because (1) 
the testing station was constantly moved due to room 
scheduling problems, (2) some of the rooms were poorly 
designed for this purpose, (3) the monitor proved to be less 
than adequate on more than one occasion, and (U) the U day 
testing interval probably provided less scheduling 
flexibility than needed because many students have multiple 
tests during the week of a standard testing period. 
Although the testing intervals were functionally short, the 
students made the most of them and distributed themselves 
over the days (item 7) for various reasons: other tests 
(79%, item 3) , study time (59%, item 9) , personal reasons 
[35%, item 11), and illness (3%, item 10).. 
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While the solutiori does not require a test- re test 
patrern of testing, thi5 is an example of a pattern that 
could not be readily us^d with standard testing procedures. 
Thus, the suudents* acceptance of and performance under this 
pattern were measures of the solution. Acceptance of the 
retest option was high the average student took 1.8 of 
the 3 possible retests (2o%, 33%, and 32% of the 'right* 
class took retests at 1 , 2, and 3 testing periods, 
respectively). 88% of . th^ iretesters (item 16) believed they 
learned more as a result of retesting, and the average 
retester significantly impjroved his score (for the 'right' 
class the average score <^hanges for the 3 testing period- 
were s.e. ^ 1.3; lif., s.e. = 1,3; and • - .99). 

Retesters improved their scores whether they studied as 
usual for the first test or studied in anticipation of the 
retest. An estimated i\9^ (based on surveyed students who 
gave their IDs and marked item 15) of the retesters claimed 
that they studied as u^ua^ for the first test, and they 
increased their score by 13% (s.e- = 3.2), One member of 
this group calculated that her class grade under a standard 
testing procedure would i^ave been her usual C, but by 
correcting weaknesses observed in the first tests and taking 
all r^t^sts^ har final cjr^^^ was an A. An estimated 51% of 
the retesters 'cJLaimed th^t they studied in anticipation of 
the retest, ajid this group increased its score by an 
estimated M% f':s.e. = 2.0). 

Lh:-^d on conversations with stuients, I believe that 
those anticipairing the i^e^^st can be divided into at least 3 
groups: (1) those who found it 'inconvenient' to prepare 
for the first t&st, (2) those who had genuine scheduling 
problem- not solved by tne 4 day testing interval, and (3) 
those who found that Practicing on the first and 
concentrating on the second tesi: was an effective method of 
study. Theoretically^ ^ student maximizes his grades by 
preparing for tiia first test ^d viewing the second as a 
^backup', but many of th^ • practice-concentrate* students, 
who were 'good* students/ believed that their method was 
best for them. Thus, tb^ student's reactions to the test- 
retest pattern were highly ^ariei and personalized^ and the 
vai-ious personalities se^^^d to use it to benefit their 
educations. 

In closing tiis secti^n^ I restate that the reaction to 
IT3 was overwhel2i-ngly positive. This fact is summarized in 
the following: (T) the grade shifts of retesters (table 2), 
(2) the 93% preference for ITS (item 18, figure 16) /and (3) 
the student^s positive com^^nts (figure 17). 
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Edu cational B^SEects^ ISaative 

Possible negative aspects of ITS were surveyed ^ith 
items 21 and 22 of figure 16. 1656 (item 21) of the stt^^^^its 
felt that the^s were substantial variations in difficulty of 
the tests. Having seen these tests^ r do not believ^ this 
to be true. In any case, the following experiment w^^ done 
to check the systems ability to generate equivalent tQ^asnj;e 
tests. Th« same item, files Used to generate t'\f? ^^P^ct 
tests were u5^ed to generate 2 v.> • 2 selQctioti^ ^A.ih 

25 items each. The 2 3^. ^^^c ^xom^ liuu only 3 commor^ it^ms 
TMith the sdnta true responses. Copies of the tests wet<5 u^ed 
for inclass testing (Brian Davis* Genetics, Fall 1976)^ T^he 
average score for the 212 students was 17.66 (s.e. 0*28)^ 
the average srores for the 2 Versions of selection 1 v^^e 
18.45 {s.e. ^0.53) and 17.2-1 (5-^. = 0.51) and tho^e Jfor 
the 2 versior.:s of selection 2 were 17.39 (s.e. = 0.$-/) ^nd 
17.54 {s.^.^ - 0.58). This experiment does not indici^at^ a 
difference iifficulty between the selections. Thu^# the 
16% was pro:3ably an estimate of frustration or run^c^r (at 
most a. stixd^^t saw 2 tests) rather than a measujre of 
variation in test difficulty. 

51% (I5-: + 36%, item 22) of the students believ<^cl that 
the 2 week testing period interfered with learnii^^ r\ew 
material tc ^onie degree. Thi3 response is difficult to 
interpret b^^ause ITS does not regiiire such an interf er^^^e. 
Perhaps the 57% is a measure of the student^s problems in 
scheduling nh^^t study time. 

The n-^ga-rive comments are in figure 18. While j 
sympathize wit^ the students' desire to have the instf^^tor 
present at th^ testing station and while the solution does 
not require that the instructor or his aids be absent, j 
believe that testing is better served if the student is •on 
his own'. Thi2 decreases noise and confusion in the testing 
room and eliinii^^tes the advantages that might otherwise be* 
gained by the ^^re persistent students. 



C ost Effectiveness 

Though little quantitative data exist, the folxowing 
ODservations can be made: (1) the secretaries spent ^^ss 
t±m-a ennerinj items than they wo^ld have spent typi^^g the 
tescs, (2) the rooms used as testing stations woul^i have 
ozherwise been unused^ (3) the instructors 'gained* a cila^s 
period at each testing period, (a) monitors were relati^^Iy 
cieap (we used a teaching assistant, but a work 5tudy 
scident could he used because trustworthiness is thp only 



21 



Individualized Testing System Calhoun 21 

Ifnpl^mentation cont 



needer^ skill) , and (5) tests cost from U to 20 computer 
cents per copy. k careful accounting woald probably shoi/ 
that this implementation was cheaper to administer than 
standard testing procedures even tS 'igh tiii^ inipleiP - »-ation 
had start-up costs, comd not ta; ^ Ivant^^^ of , and 

employed computer p- grams (proto-y:^ ' of t 'Hscu^^sed in 

this tapor^ ^ th^ , ^ - r.^ lativoly in-^j, Uci^nr awkward to 

use. 
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This report has dealt with the solution at the level of 
the individual class.- It show^ that all problems inherent 
in standard -multiple choice testing are addressed by the 
solutionr and it shovs that th© solution is popular with 
students for educationally sou^d reasons. However, the 
benefits of testing stations with sets of unique, equivalent 
measure tests (not necessarily all multiple choice) could 
have many 'higher' applications. Following are 3 
possibilities, 

(1) Students could readily obtain credit (even grades) 
for a course by taking the final for that course. The 
departments would encounter tidditional expense, yet 
students using this method would be subject to the same 
standards as students who tooX the course* Precocious 
entering freshmen^ some transfer students (those with 
certain credit transfer problems), and some bright upper 
classmen might eliminate sey^^ai lower level course 
reguirementSe allowing them to complete college sooner or to 
take more free electives. This inethod might be especially 
rewarding if current irem fiias and study guides were 
maintained at high school libr^i^i^Sr and testing stations 
were available at freshman orientation* Perhaps students 
from 'better' high schools could eliminate much of the 
freshman year requirements. 

I realize that students cB^ix presently exempt certain 
classes by examination* However/ the present procedure does 
not ensure that the student has the specific facts needed to 
complete subsequent courses of ^ i^equired series* 

(2) With little or no effort* departments could require 
graduating seniors to take a spectrum of their tests- The 
test results, coupled with tho^^ of appropriate national 
tests, would be a measure of th^ students' total education, 
and thus shoald help the depa ftm^nts find weaknesses in 
their progra^ma, 

(3) Cla^sses with multiple s^<=^tions could be readily and 
evenly tested^ and the relative strengths and weaknesses of 
the instructors could be readily ascertained. These 
instructor diagnostics would much more meaningful than 
those presently obtained by VPfS studen*^ survey method as 
these would be based on actual student performance. 
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In addition to the • higher' applications resulting from 
a testing station with sets of unigue, eguivalent measure 
tests^ the item file design with its emphasis on 
corapactnGssr organization^ and response analysis should aid 
in the production of high quality testing material. The 
emphasis on publication should encourage distribution of 
this material. Indeed, authors of this material could find 
themselves in a market situation similar to that of text 
books. Thus, the more talented producers of item files 
would be rewarded for their efforts, and the entire 
education community would share the benefits at a nominal 
cost. 

In clbsing this report, I stress that I have listed 
only a fe;jj 'higher' applications of the solution. No doubt 
the solution has many other far reaching applications as the 
production of large sets of high quality, unigue, eguivalent 
measure tests reduces most time-space-effort constraints on 
testing • 
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Figure 1 



•TWO SAMPLE ITEMS FOR A DEMONSTRATION OF THE COMPUTER PROGRAMS 
•Item from a genetics item file of 550 items. 
: 139.557 

The following/ listed in order of dominance, are 4 alleles in rabbits: ct 
colored; c(ch), chinchilla; c (h) himalayan; ' 
c, albino* a colored rabbit crossed to a colored rabbit produced 3 colored 
chinchilla. 

The parental' genotypes could have been 









c (ch) X c^- c (ch) 


^2 


c^ 


c (ch) X c+ c (h) 


-3 


c+ 


c(ch) X c(ch) c(ch) 


-4 




c+ 


c (ch) X c (ch) c (h) 


-5 




c + 


c (h) X c (ch) c (ch) 


♦ 6 




ct 


c (ch) X c 


-7 




c + 


c X c (ch) c (ch) 


-8 



none of the other answers are correct 
.Item from a physiology item file of 2^2 items- 
: 159.5 

An amino acid residue that contributes to ll bonding between amino acid side 
chains (R groups) 

♦ 1 
serine 

♦ 2 

tyrosine 

.comments can be placed before any response 

♦ 3 

threonine 

♦a 

aspartic acid 

♦ 5 

glutamic acid 
-6 

proline 
-7 

tryptophan 
-8 

isoleucin e 
-9 

leucine 
-10 

valine 
•11 

alanine 

-12 25 

glycine 



~' O el. A raw file with 2 items, 139.557 and 159.5. Note that column 
ERXC^^'^^^ved for special characters. 
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-TWO SAMPLE ITEMS FOR A DEMONSTRATION OF THE COMPUTER PROGHAKS 7610 0 423 4915000 
.Item from a genetics item file of 550 items. 
139.55700000 0 3 5 22 

The following, listed in order of dominance, are 4 alleles in rabbits: c+, 

colored; c (ch) chinchilla ; c (h) himalayan; 

c, albino. A colored rabbit crossed to a colored rabbit produced 3 colored- 1 
chinchilla. 



The parental genotypes could have been 
+ 1 0 
c+ c(ch) X c+ c(ch) 



+ 2 0 

c+ c(ch) X c+ c(h) 

3 0 

c+ c(ch) X c(ch) c(ch) 

4 0 

c+ c(ch) X c (ch) c (h) 

5 0 

c+ c(h) X c (ch) c (ch) 

♦ 6 0 

c+ c(ch) X c+ c 

7 0 

c+ c X c(ch) c(ch) 

8 0 



0 
0 
0 
0 
0 
0 
0 
0 



0 
0 
0 
0 
0 
0 
0 
0 



0 
0 
0 
0 
0 
0 
0 
0 



0 
0 



none of the other answers are correct 
.Item from a physiology item file of 242 items. 
: 159.50000000 0 5 7 28 

An amino acid residue that contributes to H bonding between amino acid side 
chains (R groups) 

♦ \. 1 0 0 0 
seri^ne 

0 0 0 

tyrosine 

.comments can be placed before any response 

* 3 0 

threonine 

♦ 4 0 
asi?artic acid 

* ' 5 0 
glutamic acid 

6 

proli ne 
7 



tryptophan 
8 

isoleucine 
9 

leucine 

10 
valine 

11 

alanine 
12 

glycine 



0 
0 
0 
0 
0 
0 
0 



0 
0 
0 
0 
0 
0 
0 
0 
0 
0 



0 
0 
0 
0 
0 
0 
0 
0 
0 
0 



0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
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r' Q 1 2. Item file FIG2 produced from raw file FIG1 by ITSFMT, FIG2 is 
iERsiC^P^^ f^^® for use by the other computer programs. 
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TWO SAMPLE ITEMS FOR A DEMONSTRATION OF THE COMPUTER PROGRAMS 761004234915000 
TRaE RESPONSES CHANGED BY 3 S FALSE CHANGED BY 5 FOR FOLLOWING ITEM 

3 'cisg.ss?' 

TRUE RESPONSES CHANGED BY 5 & FALSE CHANGED BY 7 FOR FOLLOWING ITEM 

24 'iisg.s* 

STATISTICS: 
INPUT ITEMS = 2 RECORDS = 53 

OUTPUT ITEMS = 2 RECORDS = 53 



Figure 3. ITSFMT produced this statistics listing and item file FIG2- 
srmultaneously from raw file FIGI. Since FIGl had no logical errors, 
only statistics appear in the listing. 
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Figure 4 



.TilO SAHPLE ITEHS FOR A dSMONsTR ATION OF THE COMPUTER PROGBAHS 76100 4234915000 
.Item from a genetics iteC file of 550 items. 
: 139.55700000 1273 3 5 22 

The following, listed in order of dominance, are 4 alleles in rabbits- c+. 

colored; c (ch) , chinchilla; c (h) himalayan; 

c, albino. A colored rabbit crossed to a colored rabbit produced 3 colored- 1 
chinchilla. 

The parental genotypes could have be^^n 
+ 1 85 

c+ c{ch) X c+ c(ch) 
+ 2 84 
c+ c(ch) X c+ c(h) 

- 3 190 
c+ c(ch) X c{ch) c(ch) 

4 219 
c+ c(ch) X c (ch) c(h) 

- 5 201 
c+ c (h) X c <ch) c (ch) 

+ 6 89 
c* c(ch) X c+ c 

7 2 05 
c* c X c (ch) c(ch) 

8 204 
none of the other answers are correct 

.Item from a physiology item file of 242 items. 
: 159.50000000 I50O 5 7 2b 

An amino acid residue that contributes to H bonding between amino acid side 

chains (R groups) 



339 


6715 


15255 


316 


6804 


14536 


48 


1 1400 


2160 


51 


13578 


2193 


5^^ 


1 1859 


2376 


360 


7298 


15480 


45 


12195 


1935 


56 


8832 


3584 



+ 1 


120 


180 


7440 


9000 


serine 










+ 2 


130 


195 


8190 


9555 


tyrosine 










.comments can be 


placed before 


any response 


+ 3 


1 10 


165 


6710 


8415 


t hreonine 










♦ 4 


58 


232 


4350 


11600 


aspartic acid 








+ 5 


62 


248 


4464 


12400 


glutamic acid 








6 


192 


22 


9792 


1 078 


proline 










7 


154 


66 


1 1088 


1980 


tryptophan 










8 


180 


20 


9360 


980 


isoleucine 










9 


212 


23 


1 081 2 


1 104 


leucine 










ID 


165 


29 


8250 


1450 


valine 










174 


38 


8700 


1900 


alanine 










12 


182 


43 


8736 


21 93 



glycine 



Figure 4. Item file FIG2 with simulated use-statistics. 
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Figure 5 

TWO SAMPLE ITEMS FOR A DEMONSTRATION OF THE COMPUTER PROGRAMS 76100U23U915000 

TWO SAMPLE ITEMS FOR A DEMONSTRATION OF THE COMPUTER PROGRAMS 761004 234915000 

Item from a genetics item file of 550 items. 
: 139.55700000 1273 3 5 

Item from a physiology item file of 242 items- 

: 159.50000000 1500 5 7 ^. . 



Figure 5. Contents listing for item file FIGU. 
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Figure 6 



FCPHA, FORD CALHOON, INTRO PHYSIOLOGY, BIOL 3111 76092 1 U01 05000 

FCPHA, FORD CALHOON, INTRO PHYSIOLOGY, BIOL 3111 76 0921140 105000 

AQOEODS SOLUTIONS 



!• structure of water 










: 1. 00000000 


0 








: 2. 00000000 


0 


■J 






: 3. 00000000 


0 


1 

1 






: 4. 00000000 


0 


5 




2. chemical equilibria 










: 5.00000000 


0 


2 


7 




: 6. 00000000 


0 


3 






: 7. 00000000 


0 


1 
1 




3. pH calculations 








3. 1. 


strong acids added 










: 8.00000000 


0 


-J 






: 9.00000000 


0 








: 10.00000000 


0 


- 






: • 11.00000000 


0 






3.2. 


strong base added 










: 12.00000000 


0 


- 


/i 




: 13. 00000000 


0 


- 


/I 




: 14.00000000 


0 




/I 




: 15.00000000 


0 






3.3.' 


compute IH+I 










16.00000000 


0 




a 




17. 00000000 


0 




a 




: 18.00000000 


0 




a 




19.00000000 


0 




a 


3. a. 


for Weak acids 










20.00000000 


0 




a 




21 .00000000 


0 




a 




22.00000000 


0 








23. 00000000 


0 




a 




24.00000000 


0 




4 




25.00000000 


0 




4 




26. 00000000 


0 




^ 4 
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FCPHA, FORD CALHOON, INTEO PHYSIOLOGY, BICT. 3111 7605,. 1 1 40105000 



3.5. 


pKb from ionization constant 








: 27. 00000000 






4 




: 10000000 


0 




4 




: 0000 00 0 


0 




4 


4. bcffeir:« 










: iCiiCOOOOOOO 


0 


2- 


4 




: JOO 0000 0 


0 


z 


If 




: 3^. 30000000 


0 


1 






: 33- 00000000 


0 


1 


If 




: 3^,00000000 


0 


1 


4 




: 35 .00000000 


0 


1 


4 




: 3(-* 00000000 


0 


b 


1 1 


THERWODYSmfi^CS 










: 3... 00000000 


0 


1 


If 




: .00000000 


0 


It 


9 




: 00000000 


0 


7 


7 




: .00000000 


0 




If 
4 




: 42.00000000 


0 


c 
J 






: 43.00000000 


0 


/. 


If 




: 44.00000000 


0 


J 


If 




; 45.00000000 


0 




4 




: 46.00000000 


0 


J 


i <t 
1 J 


CHEMICAL C021P0NENTS OF CELL 








1. lipids 










47. 00000000 


u 


c 
J 


1 J 




48.00000000 


0 


0 






49.00000000 


0 


6 


6 


2. carbohydrates 










50. 00000000 


0 


2 


4 




51.00000000 


0 




6 




52.00000000 


0 


3 


9 




53. 00000000 


0 


^ 


5 




54.00000000 


0 


2 


5 




55.00000000 


0 


3 


4 




56.00000000 


0 


4 


4 




57.00000000 


0 


5 


6 




58.00000000 


0 


5 


5 
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FCPHA, FvDHD ?JDO!J^ INTRO FtirSIOLOGY^ BT--^^ 3111 



7Se: 521 T 401 130 00. 



3* amino 







0 


1 








0 


1 


2 




• O , ». J\j ^j. 'J ^ J c 


0 


1 


2 




: 62. r^jOC^/'C:!' 


0 


1 


4 




: 63. OD^li:--u> . 


0 


1 


4 




: 6a.*oa'Br*ar: • 


0 


1 


4 




: 65...r :;uo: . 


0 


1 


4 






0 


1 


5 




: 67- o^uc^'OOVji): 


0 


1 


t: 




: 6 8,. c ' - lOoav ^ 


0 


2 






: 6 9. [ . 


0 


2 








0 


2 


4 




• 71^0 i^.- - J 


0 


3 






: 72.- - 0 


rv 

0 


1 


5 




: 73. . 


0 


1 


5 


proteins 








1 • 


s ui; uc r-UiT' 












0 




7 




i 7d • ; Q. ..• jj 0 


0 


5 


6 






0 


7 


6 




: 77, : JVOOO 


0 


6 


6 




: 78. C.aorx}.;, : JO 


0 


1 


1 9 




: 79^0OJiJC0a0 


0 


3 


7 




: 80. Oe'lDLV^ jO 


0 


a 


8 




: 81. j;.: :dojo 


0 


1 


1 9 




: 82.00JL...D.3C3 0 


0 


5 


7 




: * 83 .OOl' JvDOOO 


0 


5 


1 1 




: 8a.Q(L\.CI(D00 0 


0 


1 


4 




: 85*0 au-:/u 


0 


2 


5 




86. OOaiOv. VQ 


0 


6 


5 




87. OQfflia- j'O 


0 


3 


5 




88. 0 00}:OIC;rL:G 


0 


4 


4 




89.oo.Q:^]E:xLro 


0 


2 


5 




90. OOO/Mau; 


0 


1 


4 




91 .ooaaQQo: 


0 


4 


4 




92. 00000000 


u 


/I 


c 
D 




93. 00000000 


0 


2 


6 




94.00000000 


0 


7 


8 




95.00000000 


0 




a 




96. OO.DOC OO.G 


0 




4 




97. Oaafii^jOO 


0 




4 




98.oa)GaacQO 


0 




4 




99.00)005)000 


0 




4 




1 00. OOO'OOOJQO 


0 




4 




101.0 fliiOOCOO 


0 




4 




102*0 jOOOOO 


0 


2 


4 




1 03. 0. )or 000 


0 


4 


6 




lOU.OOJ. , 000 


0 


7 


10 
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Figure 6 cout 



FCPEA, FQED CALHOaN, INTRO PHYSIOLOGY, BIOL 31 M 76 0921 1 401 C^OO 0 

^•2. allosteric 





1 1 Ud. 00000000 


0 


2 


2 




3. enzyme nomenclature 










- 1 06 . 00000000 


0 


3 


5 




: lU/.uOOOOOOO 


0 


1 


5 




I lUo.JOOOOOOO 


0 


1 


5 




• i m U \J\J \J\J \J\J \J 


0 


1 


5 




classification 










c 110.00000000 


0 


1 


5 




: 1 11.00000000 


0 


2 


5 




: 1 12* 00000000 


0 


a 


5 


5. 


CO factors 










: 1 13.00000000 


0 


a 


4 




: 114. 00000000 


0 


3 


5 




: 115.00000000 


0 


£4 


5 




: 116.00000000 


0 


6 


5 




: 117.00000000 


0 


U 


7 




: 118.00000000 


0 


2 


5 




: 119.00000000 


0 


6 


6 



Figure 6. Contents listing for item file FCPHA. This on-line parallels 
the lecture in Introductory Physiology. 
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Figure 7 



■Wr SAMPLE ITEMS FOR k DE'MONST 



TWC. Sr!5PLE ZTSKS FOE DE -.C.MSTF/ r : OF THE C0K?UT: ^OGRAHS 76 10 34234915000 
Item . ro-. = genetic;; iteni file .- f ;.. items. 

.:,9.5570000C 127? . 5 : V following, listed in ord 

;:c .orad; c (ch) , chinchilla; 
z, albino. A colored rabbit 
c:....nchilla, 

rh-' parental genotypes could 

1 424 20 79 45 + c+ c(ch) X c+ c (ch) 

2 400 21 81 46 + r+ c (ch) X c+ c(h) 

3 238 80 60 45 - c+ c (ch) X c(cii) c (ch) 
270 81 62 43 - c+ r(ch) X c (ch) c(h) 

5 255 79 59 44 - c^ r (h) X c (ch) c (ch) 



6 



8 



424 


20 


79 


45 




c + 


400 


21 


81 


46 


+ 


r + 


238 


80 


60 


45 




c + 


270 


81 


62 


43 




c + 


255 


79 


59 


44 




c + 


449 


20 


82 


43 






250 


82 


59 


43 




c + 


260 


78- 


43 


54 






file 


of 242 


item 


s. 







c (ch) X c+ c 



7 250 82 59 43 - c+ c X c (ch) c (ch) 



.5v. OOOOOO 


1500 


5 


7 






Ar. amino acid 


1 












CHH-ins (E gro 


300 


40 


62 


5 0 


+• 


3srz:ne 


2 


325 


40 


63 


49 




ryxcsine 












. 


conjments cai. ' 


3 


275 


40 


61 


5 1 




t r.r-s anine 


4 


290 


20 


75 


50 


+ 


aispaxtic acid 


5 


3 10 


20 


72 


50 




giutramic acid 


6 


214 


90 


51 


4 9 




proline 


7 


220 


70 


72 


30 




tryptophan 


8 


200 


•0 


52 


49 




isoleucine 


9 


235 


9'0 


51 


48 




leucijie 


10 


194 


■ 5 


50 


5 0 




valine 


11 


212 


--.2 


50 


50 




alar, ine 


12 


225 


■~ 1 


48 


51 




glycine 



of the other answers ar 



that c 



placed before 



Figure /. ftnalyt^-^ li--ing of -xea file FIG^. ^'Thla is a -runcated 
versx.on of t.he ac~ii^ 15_ charac-er listing, truncati.^n allowed easier 
reprodmrtion of thi_:: r^poz^. ) 
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Figure 8. 



JELECTIONS 2:\":TiSZ0US ^ 2; 
LTSCi^rilE - , TEST TELE - FIG'O; 

RBQJEST = 2 10. -200; 
L:rN£SlZ3 = 80; 

TITLE = SPECIFrc :I0N E2:.\?1PLE; 
END; 



.figure a. SpeczL ficatior.s for a set of U tests (2 versions 
of each of 2 seL^ctions) , 
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Figure 9 



76 1007200919, S?ZCZ?ICATIO:;s SOUGHr 

SELECTIONS = 2; 

VERSIONS = 2; 

ITEH^FILE = FIG--'. 

T£5T_FILE = FIG /; 

REQUEST = 2 'OC -2^. : 

LINESIZE = 8 ; 

TITLE = SPEC FI:AT:. ' EXif5?LE: 
END; 

76 100720091- , 5?E CnnCATIQ.NS OBTAINED 

TITLE = SPE-CrFICiCTTC:; EjCAf5PLE 

TEST_FILS = 7IG1C 

ANALYSXS^FILZ = 

K£Y_PSINT_FTLE = 

KE-Y_PTJNCH_FILE = 

SELECTIONS = 2 

VERSIONS = 2 

FrR.ST_KEY = 1 

LISESIZE = 30 

PASES IZE = 65 

TOP_LINE = 1 

select#riget = 
sel.ect#wroe;g = 
rai^tdom numezr = 



761 0072009' 

HEQ7I^-1 F 



Of) ' 



1 

U 

26091 

:ZBL2 ITE^S 
ITEMS 
100.000DQ0O0 



2c oooooooo 



761O0720091 , AV^^IIABILITY CF REQUE3T"3D ITE.MS 
REQ-JEST . CIESTDD ?AVAXLA.BLS 

1 2 



/•61O072t.'^ . 93: Sr^SCJFICATIONS ^CHTGHT 
GliQUBS i>£G.CL*^:5EID ^ 1 



Fig:ire , -.ISGZN ope:irated on speciiiica tions of f^rrure 8 to 
rxo::uce -his specifications listing a^^.id tests of figure 10. 
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Figure 10 



75100720091 9r SSLECTIOIi 1^ VEHSrO!^ 1, KEY 1^ 4ITEK5 2 
SPECIFICATION EXAMPLE 

1- The followir>(j, iistevi in order of dominance, are U alleles in rabbits: 
c^, colored; c [ch) , chinch:^iZ.a; c(h) h:_iiiiaayan; c, albino, A colored rabbit 
cross.ad to a colored rabbit: produced 3 cj:lorad: 1 chinchilla. The parental 
genotypes could have been (1) none of tzi^e orher answers are correct 
C2) c* c(h) X c(ch) c(ch) (3) c+ c (ch c+ c(h) (4) c+ c (ch) X c (eh) c (ch) 
(5) c* c :x c (ch) c (ch) 

2» An amino acid resia::u€ that cont L::j3Ui^s to H Donding between aflsino acid 
side chains (B groups) (1 proline (Z) alanine f3) threonine 
(4) valine (5) i30leuc:-.n-e 



761007200919, SELECTION 1, VZjRSION 2r 't-IT 2, #I!I'E^^S Z 
SP3CIFICATI0N EXAMPLE 

1. The following, lis*^ ^ 1 In orcier cr ramin^:.r5^ 
c^e colored; c (ch) , chinch^^^lx; (h' hi-^c-.^d^an; c, 
crossed to a colojred rabbi-: r:rouuc;a_: 3 c;..;ored: 1 cr:r. 
geaotypas could h- v^ been ' n c :c cr (-n) c{ch) 
(3) c+ c(ch) X d-ii) c(ch) ^ ) :* c( :rh) - 
(5) none of the c-rher aG.5v'-?r^ are c-:rT^/r- 

2. An amino dzii r^^sirl . *_nat r:3n'^r_jutes tc . r^rnding between amino acid 
side chains (R groups) (T, '::reon-::-^ [:ZJ isjoieirr-in-^ (3) valine 

(U) alanine (5) proline 



.zre 4 alleles in rabbits: 
- .IruD. A colored rabbit 
^.:chiLlla, The parental 
c+ c (ch) X c+ c (h) 
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Figure 10 cont 



76100^00919, SELECTION 2, VERSION 1, KEY 3, LITEMS 2 
SPECmCATlON EXAMPLE 

The following, listed in order of dominance, are a alleles in rabbits- 
c+, colored; c (ch) , chinchilla; c(h) himalayan; c, albino. A colored rabbit 
crossed to a colored rabbit produced 3 colored: 1 chinchilla. The parental 
genotypes could have been (1) c+ c (ch) X c+ c (2) c+ c (ch) X c (ch) c (ch) 
(3) c+ c(ch) X c (ch) c(h) (4) none of the other answers are correct 
(5) c+ c X c (ch) c (ch) 

2. An amino acid residue that contributes to H bonding between amino acid 
side chains (R groups) (1) proline (2) glycine (3) asparticacid 
(t) alanine (5) valine 



761 007z.00919, SELECTION 2, VEBSION 2, KEY 4, #ITEMS 2 
SPECIFICATION EXAMPLE 

1. The following, listed in order of dominance, are 4 alleles in rabbits- 

c+, colored; c (ch) , chinchilla; c (h) himalayan; c, albino. A colored rabbit 

crosse-: to a colored rabbit produced 3 colored: 1 chinchilla. The parental 

genotypes could have been (1) none of the other answers are correct 

(2) c+ c(ch) X c(ch) c(ch) (3) c-i- c X c (ch) c(ch) (U) c* c (ch) X c(ch) c (h) 
(5] c+ c(ch) X c+ c V i V / 

2. An amino acid residue that contributes to H bonding between amino acid 
side chains (R groups) (1) glycine (2) valine (3) aspartic acid 

(a) alanine (5) proline 



Figure TO. ITSGEN operated on specifications of figure 8 to produce 
these a tests (2 versions of 2 selections) and the specifications iistinq 
of figure 9. ^ 
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Figure 11 



KEY_PRINT_FILE = FIG14; 

ITEM FILE = BDGNA; 

REQUEST = 5 5 12 17 23 31 41 ; 

ITEM^FILE = 3DGNB; 

REQUEST = 3 97 111 112; 

ITEK_FILE = BDGNC; 

REQUEST ^ 6 118 135 l«i8 165 175 184; 
ITEH_FILB = BDGNF; 
REQUEST = 3 417 436 449; 
TE5T_FILE = FIG13; 

TITLE = FIRST SAttPLE TEST^ GENETICS^ FALL 1976: 
END; 



Figure 11* Specifications for 1 test. 
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Figure 12 



760928232541, SPECIFICATIONS SOUGHT 

KEY_?BINT_FrLE = FIG14; 

ITEM^FILE = BDGNA; 

REQUEST = 6 5 12 17 23 31 41; 

ITSM^FILE = BDGNB; 

REQUEST =3 97 111 112 ; 

ITEM_FILE = BDGNC; 

REQUEST = 6 118 135 148 165 175 184; 
ITEM^FILE = BDGNF; 
REQUEST = 3 417 436 U49; 
TEST^FILE = FIG13; 

TITLE = FIRST SkULPLE TEST, GENETICS, FALL 1976 
END; 

760928232541, SPECIFICATIONS OBTAINED 

TITLE ^ FIRST SAMPLE TEST, GENETICS, FALL 1976 

TEST_FILE = SAMPLE 

ANALYSIS_FILE = 

KEY_PRINT FILE = KEY 

KEY^PUNCH^FILE = 

SELECTIONS = 1 

VERSIONS = 1 

FIRST_KEY = 1 

LINESIZE = 132 

PAGESIZE = 65 

TOP_LINE = 1 

SELECT#RIGHT = 1 

SELECT#lJRONG - 4 

BANDOM_NUMBER = 9281 
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Figure 12 cont 



760928232541^ 
REQUEST FILE 



POSSIBLE ITEMS 
ITEMS 





BDGNA 


^ nnnnnnnn 

^ m \J\J\J\J\J\J\J\J 




BDGNA 






BDGNA 


17 on on n n n n 

1 im\J\J\J\J\J\J\J\J 




BDGNA 


0 oooo c\c\ c\r\ 




BDGNA 


11. ooooonnn 




BDGNA 


ui . ooooooon 


2 


BDGNB 


Q7 n^finnnnnn 


2 


BDGNB 


111-000000 0 0 


2 


BDGNB 


112. 00000000 


3 


BDGNC 


118. 00000000 


3 


BDGNC 


135. 00000000 


3 


BDGNC 


1 48. 00000000 


3 


BDGNC 


165. 00000000 


3 


BDGNC 


175.00000000 


3 


BDGNC 


184.00000000 


U 


BDGNF 


417. 00000000 


4 


•BDGNF 


436. 00000000 


4 


BDGNF 


449. 00000000 



760928232541, AVAILABILITY OF REQDESTED ITEMS 
REQUEST #REQUESTED ^AVAILABLE 



1 
2 
3 
U 



6 
3 
6 
3 



6 
3 
6 
3 



760928232604, SPECIFICATIONS SOUGHT 
GROUPS PROCESSED = 1 



Figure 12. ITSGEN. operated on the specifications of figure 
11 to produce this specifications listing and the test'Of 
figure 13. 
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Figure 13 



760928232541, SELECTIOH 1, VEHsiON 1, KEY 1, #ITEMS 18 

FIRST SAMPLE TEST, GENETICS/ PALL 1976 ~" 

l\ "^"'^^l'^ I-^" °f Independent Assortment states that each parent passes on 

(2) 9:3:3:1 ratio (3) random chromosomes (4) heterozygosity (5) random combin 

2. In peas, smooth seed (S) is dominant over wrinkled (s) and yellow seed f 
yellow seeds is testcrossed- The 100 progeny plants are all smooth yellow, what 

(1) ssYy (2) SsYY (3) SSY/ (4) insufficient information (5) SSYY 

...» '^^^ environment influences (i) no human traits (2) most characteristic 

(4) only non-mherited traits (5) only a few traits 

t' , epistatic to b and b, what phenotypic ratio will result from th 

(3) 1:1:1:1 (4) 9:6:1 (5) 13:3 

5. A man and wife, are Ijoth heterozygous for a recessive autosomal gene whic 
probability that the first chxia will be a girl with phenylketonuria? (1) 1/4 

/o. ^' ^ chi-sguare test yields p of 0.001 the hypothesis is (1) correct wi 

(2) guestionable and should retested (3) incorrect (4) incorrect with a pro 

7. All cells in each individual of a species have the identical number of 

(4) ribosomes (5) chromosomes 

8. Growth in higher organisms occurs primarily by (1) mitotic cell divisio 

(5) cytoplasmic expansion 

9. A diploid cell has two metacentrics and two acrocentrics. After one mito 
metacentrics and how many acrocentrics? (1) 2 and 0 or 0 and 2 (2) 2 and 1 or 

10. Stage of meiosis in ^^ich homologs are paired (1) anaphase II (2) telo 

(5) anaphase II 

11. A human cell in G1 has about 6e-6: micrograms of DNfl. How many micrograms 

(3) 3E-6 (4) 6E-6 (5) 24E--6 

12. Microsporogenesis differs from megasporogenesis (1) 2 meiotic divisions 
(3) takes place in flowers more than one nucleus in cell at fertilization 

13. Meiosis I nondisjunction of sex chromosomes in a human male would -lead t 
(2) no sex chromosomes (3) 2 X's (4) i x (5) 1 Y 

14. A man has ichthyosis^ an X-linked recessive. His wife is homozygous norm 
(1) 1/4 of children (2) none (3) 1/2 of children (4) all daughters (5) 1/2 s 

15. Claret-eyed female fUes are crossed to red-eyed males. All progeny are 
(1) X-lmked recessive (2) autosomal dominant (3) X-linked dominant (4) autos 



Figure 13. ITSGEN operated on specifications of figure 11 to produce 
this test. (This is a truncated version of the actual 132 character, 18 
item test: truncation allowed easier reproduction of this report.) 
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Figure 1U 



760928232541, SELECTION 1, VERSION 1, KEY 1, tITEMS 18 

(1) 5 (2) 5 (3) 2 (4) 1 (5) 4 (6) 1 (7) 5 (8) 1 (9) 3 

(10) 3 (11) 2 (12) 2 (13) 2 (14) 2 (15) 2 (16) 5 (17) 2 
(18) 1 



Figure 14, Key for test of figure 13. 
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TESTING STATION PEOCEDURE 

Report to the testing station with a #2 pencil^ picture ID 
(student ID, driver's licence, etc.) f and other materials 
required by your instructor. You will not be admitted 
without these materials. Under no circumstances will 
scratch paper, books^ notes, pets, or other non-required 
material be allowed in the testing room, 

INITIATION OF TEST 

1, You must show the monitor a picture ID, He will check 
it against the class roll and loan you a test, 

2, Write your name at the top of the answer sheet. 

3, Black out your student ID on the answer sheet (item 1), 

4, Do all scratch work on the tost, 

5, Consult the monitor for technical advice. The m.onitor 
will not answer any questions concerning the subject matter 
or wording of your test, 

SPECIAL CODES AMD INSTRUCTIONS FOR SOME CLASSES • 

6, Exponential numbers ..rare in E notation; e,g,, 1,3 times 
ten to the minus five is "~l,3E-5, 

7, When answering items requesting 'second digit only', use 
only the second digit of your computation; i,e,, starting at 
the left and ignoring leading zeroes, take the second digit;- 
e.g., 

if your computation yields 903 75, your answer is 0, 
if your computation yields l,987E-4, your answer is 9, 
if your computation yields 1,0, your answer is 0, 
if your computation yields ,00567, your answer is 6, 
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Figure 15 cont 



ON COMPLETION OF THE TEST 

8. Have the monitor exchange your ansver sheet for your 
test key, 

9. Return to. your seat and study your test. This is your 
only chance to study your test cis it will be destroyed. 

10. Please note any technical problems on the back of your 
test. Misspelled words, difficult phrases or terms, and 
multiple correct answers are examples. Also specifically 
note other short-comings of the test; e.g., no questions on 
subject X, too many questions on subject Y. please make the 
monitor aware of these notes when you turn in the test so 
that the monitor will route your test to your instiructor. 
Any notes you make may improve subsequent tests. 

11. On completion of your study and before leaving the 
room, you must return the key and the test to the monitor. 
Removing tests from the testing room or copying test items 
is a violation of the Honor Code. 



Figure 15. Instructions on the use of the testing station 
given to all students the week before the first testing 
period. 
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Figure 16 



Please give us some facts and opinions to help in evaluation 
of rhis course and the out-of --class testing system used for 
your multiple choice test. 

OPTIONAL Enter your ID number as usual in the upper left 
corner of the answer sheet. Your instructor will never see 
your responses. This request is made to permit correlating 
responses with course achievement. (Do not write name.) 

1. Were the multiple choice test questions appropriate for 
course content? 

17 _ _ (1) very, appropriate 

52 _ _ (2) adequate 

23 _ _ (3) somewhat deficient 

8 « ^ (4) really off-base 

2. How well were the multiple choice questions written; to 
what extent were they understandable? 

0 _ _ (1) excellent 

27 _ _ (2) good 
^5 _ _ (3) fair 

28 _ _ (4) poor 



3. 


For me 


this 


course is 




35 19 


7 


(1) very difficult 




50 60 


61 


(2) difficult 




15 19 


27 


(3) about average 




0 0 


4 


(4) easy 




0 2 


0 


(5) very easy 


U. 


The lowest- 


grade I would be satisfied with in this 


course is 








15 15 


14 


(1) A 




22 6 2 


45 


(2) B 




33 23 


38 


(3) C 




30 0 


3 


(4) D 


5. 


For me 


this 


course is 




14 13 


5 


(1) boring 




16 16 


8 


(2) slightly boring 




28 38 


39 


(3) slightly interesting 




H2 33 


48 


(4) interesting 


6. 


Rate th 


e overall quality of instruction in this course 


as 


comoared 


with 


others you have taken at Virginia Tech. 




7 5 


2 


(1) Poor 




38 45 


13 


(2) Fair 




33 42 


58 


(3) Good 




22 8 


28 


(4) Excellent 
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Figure 16 cont 



The following questions concern only the out-of-class method 
of testing used for your multiple choice test^ not the test 
questions or course. 

7. The multiple choice test could be taken over a four-day 
period. Which day did you take it? 

5 11 22 (1) first 

13 15 12 (2) second 

22 8 36 (3) third 

60 64 29 (4) fourth 

_ 2 1 (5) cannot remember 

For 8. thru 12: Various things may have influenced your 
choice of day. Tell vhether each was an influence. 

8. Tests scheduled in other courses 
87 87 79 (1) yes 

13 13 21 (2) no 

9. Desire to gain as much study time as possible 
80 74 59 (1) yes 

.20 26 41 (2) no 

10. Illness 

8 8 3 (1) yes 
92 92 97 (2) no 

11. Personal or social reasons 
42 40 35 (1) yes 

58 60 65 (2) no 

12. Gain information from someone who took the test earlier 
10 _ (1) yes 

"50 _ _ (2) no 

13. Do you think you were able to do better on the multiple 
choice rest (on the first try) as a result of choosing your 
own day? 

34 68 67 (1) yes 

29 16 11 (2) no 

37 16 22 (3) cannot say 

'14. Did you retake the multiple choice test? 
49 47 57 (1) yes 
51 53 43 (2) no 

15. Did ava.ilability of the retest influence how hard you 
studied for the first multiple choice test? 

32 37 37 (1) yes 

68 55 53 (2) no 

8 10 (3) cannot say 
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Figu« 16 cont 



16, If you retook the multiple choice test^ do you feel you 
learned more as a result? 

63 78 88 (1) yes 

23 12 7 (2) no 

14 10 5 (3) cannot say 

17. Did the atmosphere of the testing room and freedom from 
time pressure .aid your performance on the multiple choice 
test or retest compared vith that of regular classroom 
testing? 

54 81 81 (1) yes 

23 13 15 (2) no 

23 6 3 (3) cannot say 

IB, Which multiple choice testing approach do you prefer? 
29 24 7 (1) inclass tall s:tude:iits take 

the same test on tine same day) 
"'I '^6 93 (2) out-of-cLass fas ±n this cla^ 

with oppoxtunirty ro retake) 

19. ^ Sup:3o:se that the oppartuniity to retake was not 
available. Then which approach would you prefer? 

50 -2 20 (1) inclass 

50 d8 80 (2) out-of -class 

20. Rate the helpfulness of receiving the key z:mraediately 
after completing „the multiple choice test, 

43 39 57 (1) very helpful 

39 46 37 (2) helpful 

18 15 5 (3) not helpful 

21. Do you feel that there was a substantial variation in 
degree of difficulty from one person's examination to 
another's? 

^ 22 16 (1) yes 
. 22 24 (2) no 

56 59 (3) cannot say 

22. Did spreading examination time over a two-week period 
interfere with learning new material? 

_ 24 15 ( 1) yes ^ somewhat 
n(5 36 (2) yes^ slightly 
_ 6.0 50 (3) no 

23. Write any comments or suggestions not covered above on 
the back of the opscan sheet. 



Figure 16. Survey of three classes. The three columns are 
the percentage (rounded to integers) of students in each of 
the three classes that considered the item and marked the 
response. 
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Figure 17 



"An excellent system." 

"This system of testing seems exceptionally beneficial for 
courses such as genetics and physiology^ because the student 
was given time to assimilate all the material." 

"Overall^ I like the system^ and I definately perform and 
learn more through this method." 

"Testing ^orocedure in this individtialized manner was very 
effective as a learning device. It was more relaxed, a more 
efficient use of time and more considerate of the student." 

"1 think this is a great system and wish other teachers 
would do the 3ame." 

"I think the retake system of exams is a excellent system. 
Any faults it has are a result of the short time period the 
quarter syst^^i gives. i sincerely feel that I learned much 
more from thi^ course as a result ana furthermore I wish all 
my classes hc^ it." 

"I prefer thLs system of testing because it makes you aware 
of areas you are weak in and gives you a chance to correct 
your weaknesses. I found it gave me more thorough knowledge 
of the material, and that*s what I'm here for." 

"I like it. Let's do more this way." 

"Reduced pressure from not having a time factor involved 
really helped. Knowledge that a retest was available also 
aided in reducing pressures. Individual testing red,u.c^«^cl 
amount of distractions. I believe all of the responses were 
helpful in not only doing better, but it is also' a b-etter 
indications of what is really learned." 

"This new system really takes the pressure out of taking 
tests. I was able to study more effectively and did much 
better than usual. I hope this system catches on in the 
other departments, " 

"By being able to retake the exam we are able to know what 
type of questions will be asked and can be better prepared. 
Choosing our own day to take the exam avoids conflicts with 
other tests. " 

"I like the non pressurized testing procedure and the retake 
is of little concern, because I feel that I will do as well 
the first time as I would the second time when it comes to a 
'computer* test like this. The choice of day is also very 
good and allows the student to schedule the test around 
pressing engagements." 



Figure 17. Relevant positive student comments on item 23 
of figure 16. 
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Figure 18 



"Great system but (instructor) should be there to answer 
specific (course) questions." 

"Teacher should be present to answer questions on ambiguity 
of some qu.estions on the test." 



Figure 18. Relevant negative student comments on item 23 of 
figure 16. 
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Table 1 



Tajle 1. 
tests. 



Specif icati..ons for contro. 1 i ng generation of 



keyword 



range (defaults parenthern zed) , description 



£2ai£2i 2l tgst format 

LINESIZB 6a-(132). Width of test :?age in characters. 

PAGESIZE . ^40- (6b). Length of tesr~. page in lines. 

TOP^LINE (1)-30. First print lin- of test page, 

control of test identification 



FIRST KEY 



TITLE 



(1)-Y. Each test^ and all material related 
to xt, is uniquely and serially identified 
with a *KEY* integer beginning with the value 
of FIRST^KEY. 

(blank) or 1-32,767 characters* . Appears at 
the top of first ?ag^ of each, test, usually 
used zc-T s^peciai instructions or 

identif iira^icr^ A ritl^^ cannot contain a 
seralcolCHi, 



control of test it ems 



ITEM FILE 



RANDOM^NDMBER 



1- chararrte^ 
sp^srrif ied* 



item fj^ie 



name must 



be 



REQUEST 



(computed) or 5 digit odd nutaiber. Initial 
random number for pseudo random number 
generator. Two test geisrations will be 
equivalent ±f theizr RANDOM_NUMB ER 

spe:cif ications ax^e the s^une (assuming the 
same requests arre made from the same item 
files) . 

must be specified. Number of items to be 
randomly selected followed by the IDs of the 
possible if^ms; e^g., 'REQUEST = 10 17.5-39 
73 97.6 107.7-3000. 1 ; • requests a random 
selection of 10 items from a pool of items 
contain ing items 17.5 through 39^ item 73, 
irem 97.6, and items 107.7 through 3000.1. A 
REQUEST specification is associated with the 
first preceding ITEM_FILE specification. Any 
number of REQUEST specifications can be 
associated with an ITEH_FILE specification. 
In general / education is better served if 
each request is for 1 item from a homogeneous 
pool. 
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Table 1 cont 



kayword range (defaults parenthesized) , description 



S2Ill£2i 2£ iS^t items continued 



SELECTIONS 



SELECT#RIGHT 



(1)-Y. Number of tests to be generated by 
randomly selecting items and randomly 
selecting responses from the selected items. 

(1) -32,767. ^laximum number of true responses 
to be randomly selected from each selected 
i tem . 



1 - (4) -32, 767. Maximum number of false 
responses to be randomly selected from each 
selected item. 

1-7 character file name must be specified* 
ITSGEN vrites the tests on this print file. 

(1)^Y. Number of tests to be generated by 
randomizing the order of each item's 
responses. Thus, 2 versions of a selection 
are inf orma tionally identical, 

£2S.i£2i of §.il2£iliaLry i nfor mation 

ANALYSIS^FILE (blank) or 1-7 character file name. ITSGEN 
writes all item and response IDs for each 
test generated on this file for subseguent 
use by ITSUPD. 

KEY^PPINT^FILE (blank) or 1-7 character file name. ITSGEN 

writes the test keys on this print file. 

KEY^PUNCH^FILE (blank) or 1-7 character file name. ITSGEN 
writes the test keys on this punch file. 



SELECTS WRONG 

TEST_PILE 
VERSIONS 



Y must be consistent with 32^768 = SELECTIONS X VERSIONS + 
FIRST KEY. 
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Table 2 



Table 2, The percentage (rounded to integers) of the 
■right* class students who obtained the indicated pair of 
grades at their first testing period (N = 93), 



test , retest 
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